Aminet 25

home *** CD-ROM | disk | FTP | other *** search

/ Aminet 25 / Aminet 25 (1998)(GTI - Schatztruhe)[!][Jun 1998].iso / Aminet / util / sys / PoolMem.readme < prev next >

Wrap

Text File | 1998-05-05 | 22.8 KB | 564 lines

Short: Memory defragmentizer/AllocP superset Author: thor@math.tu-berlin.de (Thomas Richter) Uploader: thor@math.tu-berlin.de (Thomas Richter) Version: 1.35 Type: util/sys Requires: OS 3.0 ______________________________________________________________________________ Version History ______________________________________________________________________________ 1.35: - Added a patch for the AvailMem() function to include the bytes in the scratch list. - Added the ALLOC32 keyword. See below for caveats! You usually DO NOT want this! ______________________________________________________________________________ 1.34: - Added the SMALLHEADER keyword. A selected memory block can be set aside for the small memory blocks explicitly. - Fixed a documentation error in the Developpers.readme. Thanks, Dave! ______________________________________________________________________________ 1.33: - Removed a bug in the PoolMem shutdown code. The pre-1.33 releases did not free the scratch list completely. - Added the SCRATCHONLY keyword, it installs only the "scratchlist" of PoolMem, not the dynamic pools. - Added the PUDDLESIZE, PUDDLETHRES and SHRINKTHRES keywords to make the parameters of the small memory pool adjustable. - Rewrote a lot of the kernal routines, mostly for optimizations. ______________________________________________________________________________ To "SaferPatches" users: To be able to remove PoolMem later on safely, PoolMem must be run AFTER SaferPatches has been installed. Same goes of course for all other patches and the SaferPatches program. ______________________________________________________________________________ Photogenics: I got reports that this program crashes with PoolMem installed as soon as it is shut down. If anybody knows what's going on with this proggy, please let me know. It might be worth to check for correct memory allocation with MungWall and MemSniff. ______________________________________________________________________________ Developpers should check the "Developper.readme" file in this archive. It contains information about how to allocate and deallocate memory correctly. THESE AREN'T POOLMEM SPECIFIC INFORMATIONS! ALL OS FRIENDLY PROGRAMS HAVE TO OBEY THE RULES FORMULATED IN THIS FILE, they aren't my inventions. ______________________________________________________________________________ 1.32: - Rewrote the startup segment completely. All 1.2 compatible code, argument parser etc... removed. - Added a proper version check. PoolMem requires OS 3.0. - Added the NOMERGE option to avoid merging of FastMem of different speed. - As a result of the new argument parser, the CHIPFWD option DOES NO LONGER include the "INSTALL" option, both arguments should be given. - The PoolMem algorithms hasn't been changed, though. - Included new versions of PatchRAM and ShowMem. - PoolMem 1.31 and above works fine on PowerUp-Systems. Thanks to Andreas Kleinert for testing! However, the current version does not yet adjust memory correctly for the PowerUp system, that's still left to the ppc.lib and AllocP32. Might change in the future, though. ______________________________________________________________________________ Compatibility node: The game demo "The Final Odyssey" crashes when PoolMem is active. Tracing this error down to its roots resulted in an aparent bug in the memory deallocation function of this game that occurs, too, with MungWall instead of PoolMem installed. Since EVERY CORRECT PROGRAM is supposed to run with MungWall, I WON'T TRY to add a workaround for this bug. Just avoid buggy programs, as usual. Thanks to Dirk Neubauer for the hint. ______________________________________________________________________________ 1.31: PoolMem includes a patch for AllocAbs() to be able to allocate absolute memory from the scratch list. This *should* help to make PoolMem working with the 68040/68060/ppc library, thus, it should be possible to run PoolMem IN FRONT of these libraries, i.e. "SetPatch". Would be nice if somebody could sent me a report whether this works now or not. Additionally, new versions of the FragMeter and ShowMem program are included. ______________________________________________________________________________ Several people reported instabilities when running PoolMem. I can't reproduce any of these crashes here at my system, thus if you find a REPRODUCABLE crash, please, please let me know! This version of PoolMem includes debug information that might help you and me to locate possible bugs. If possible, report bugs by giving the routine names directly. It has been reported that PoolMem runs astonishingly slow on PowerUp boards. I've currently no explanation for this, if you've any idea about it, or know how to fix it - please report! ______________________________________________________________________________ 1.30: This version introduces a major rewrite of all critical routines, especially the replacement AllocMem() and FreeMem() routine. This might mean that either old bugs have been fixed or new bugs have been introduced. Anyways, check out for possible bugs since this is a release with a "0" in the version. This version might be much faster than the 1.2x version, due to some optimizations. WARNING: Since the new 1.30 release of PoolMem replaces the Exec memory functions completely, YOU MUST RUN POOLMEM IN FRONT OF OTHER MEMORY PATCHES. ESPECIALLY, RUN POOLMEM FIRST, MUNGWALL AFTERWARS or Mungwall will be non- functional. _____________________________________________________________________________ 1.22: PoolMem merges now automatically all adjacent memory lists of the same type. ______________________________________________________________________________ NEWS FLASH: Found serious bug in XiPaint that caused crashes with PoolMem, check below for details. ______________________________________________________________________________ 1.21.1: Included a new version of PatchRAM (check PatchRAM.readme for details), and a new version of the FragMeter program. _____________________________________________________________________________ 1.21: PoolMem flushes now the memory as it should if called by AllocMem(-1,..). Except that, nothing changed. _____________________________________________________________________________ 1.20: Added the CHIPFWD command. Works like INSTALL but allocates chip mem in standard (forwards) mode instead of backwards mode. Might solve strange compatibility problems. Added more sanity checks, included PatchReplyMsg to create warnings for broken programs. ______________________________________________________________________________ 1.19.1: Changed the name of the DefragMeter to FragMeter because that makes more sense. Added a chunk count to this program. PoolMem itself unchanged. ______________________________________________________________________________ About PoolMem: If you run a lot of programs without resetting the system, you'll usually find that the main memory of the computer is getting "messed up", split in lots of tiny memory "snippets" that are more or less useless due to their tinyness. It may happen that you can't start an application even though enough memory is available - because this memory is too fragmentated to be of any use. That's the point where PoolMem tries to help you: It manages the main memory in a way such that it can't get fragmentated too easely. It replaces also the function of AllocP, which is therefore obsolete. ______________________________________________________________________________ Installation and Usage: Copy the PoolMem program in this archive to the C: directory of your system partition. Add the following line to your startup-sequence: PoolMem INSTALL >NIL: You might also want to install "PatchRAM" which is included in this archive, too. It modifies the system RAM disk in a way that avoids memory fragmentation, too. Another bonus is that the RAM disk will show its true size from now on, i.e. it won't be no longer 100% full. To remove PoolMem later on, open a shell window and enter PoolMem remove The complete command template: HELP/S,REMOVE/S,INSTALL/S,CHIPFWD/S,NOMERGE/S, SCRATCHONLY/S,PUDDLESIZE/N,SHRINKTRHES/N,PUDDLETHRES/N SMALLHEADER/N,ALLOC32/S HELP: Prints a small overview of the available options. REMOVE: Remove a previously installed version of PoolMem. INSTALL: Install the PoolMem patch. CHIPFWD: Don't allocate big chunks of chip memory in reverse direction. NOTE: Starting with 1.32, this is no longer a stand-alone option, INSTALL must be given, too. NOMERGE: Do not attempt to merge adjacent memory blocks of the same type. Even if the memory types are equal, the physical characteristics of the memory blocks might differ (e.g. 32 bit memory vs. 16 bit memory). Future versions of PoolMem might respect this. fragmentation. SCRATCHONLY: Do to build memory pools for small block allocations, but use the scratch list only. PUDDLESIZE: Set the "puddle size" for the small block memory pool. If an allocation from the small memory pool fails, PoolMem attempts to enlarge the pool by this number of bytes. Defaults to 8192. PUDDLETHRES: Set the "puddle threshold" for the small memory pool. Allocations larger than this size are taken from the large memory pool instead of the small memory pool. SHRINKTHRES: Set the "merge threshold". If more than PUDDLESIZE + SHRINKTHRES bytes are unused at the end of the small memory pool, the pool shrinks and the free bytes enter the large pool. Defaults to 4096. SMALLHEADER: If this option is given, set the memory block of this index aside and use it as small memory pool. The argument is simply the number of the block in the memory list, counting from zero, and can be found by watching the output of ShowMem. The topmost memory block is number 0, the next below 1 and so on. This memory block MUST BE fast memory. If this option is given, PoolMem does not try to split any fast mem header, only the chip mem is split. The small memory block is sorted in just before the chip memory, but is used for the small memory block allocations as long as possible. ALLOC32: Align all memory allocations to 32 byte boundaries. ______________________________________________________________________________ ALLOC32 Caveats: This keyword keeps memory allocations aligned to 32 byte boundaries, which might be useful if dealing with PPC hardware - by using this keyword, the PPC cachelines can't overlap with the MC68K series cacheline. HOWEVER, using this keyword fragmentizes the memory really badly because of the unuseful unaligned boundary snippets that neither can be allocated, nor enter the scratch list. Even though memory returned from AllocMem() IS aligned to 32 byte boundaries, AllocVec'd memory isn't. It is still only long word aligned because the first longword of the vector is used to keep the size of the allocated memory. This doesn't hurt for the purpose this keyword was invented for; it is, even more, REQUIRED to work like this because the vector size should be either in the PPC or in the MC68K cache, but not in both. Due to the heavy fragmentation of memory when using this keyword, it is recommended to use it only if absolutely necessary. The standard PPC system software deals with the problem of overlapping caches correctly so there's by default no need for this keyword. This keyword works quite different from AllocP32. It doesn't use "magic cookies" in any way and is written in a way compatible to the - rather ugly - layers.library allocation methods (check the developer.readme file in this archive for details on this). The drawback of this compatibility is the rather high fragmentation, it's even higher than without PoolMem. You have been warned! ______________________________________________________________________________ Compatibility: There are a couple of relatively deep patches of the memory system that shouldn't be run together with PoolMem: - SetPatch: The 68040 library which is loaded by this program seems to allocate the MMU tables in a strange way incompatible to the scratch list of PoolMem. Easy solution: run PoolMem after SetPatch. This compatibility problems *should* have been fixed with the 1.31 option, but hasn't been tested since I've a '030 on board. - MemSniff: (The THOR MungWall replacement) Patches deep into the memory allocation routines, too deep for PoolMem. I guess that's all for now... Buggy programs that crash with PoolMem installed: - XiPaint 3.2 (or XMouse, to be precise): The message handling of the mouse driver is broken. After a long debug session deep into the night, I found that the crashes were created by the mouse driver XMouse. It replies messages that are no longer valid and have been FreeMem()'d before. THIS IS DEFINITELY ILLEGAL AND NOT A POOLMEM PROBLEM. To help you checking other programs as well, I included a tiny patch "PatchReplyMsg". You shouldn't install it permanently, though, because it slows down the message system. If, after installation of this patch, a program crashes with guru 0x01000001f, THEN THIS PROGRAM HAS THE SAME BUG AS XIPAINT and replies illegal messages. This will crash your system anyways, with or without PoolMem. Please sent a bug report to the author, or remove these programs. As I said, THIS IS NO POOLMEM problem. The guru 0x01000013 is rather typical for these programs - they invalidate the internal scratch list of PoolMem. - The Final Odyssey: The memory deallocation routine of this program is buggy. It crashes, too, with MungWall installed. Since EVERY program that crashes with MungWall is NOT supposed to be system conformal, I WILL NOT add a workaround for this problem. Additional node: All PoolMem versions so far contain workarounds for: o) Programs that rely on a "correctly" set zero-condition code of AllocMem() ("PixMate") o) the FFS/OFS ACTION_CLOSE bug that does not set the correct result code. This is currently done by the PoolMem FreeMem() replacement routine. However, the author of the FFS ("Heinz") has been informed, so it's likely that this error will be fixed in the next FFS release, probably for the FFS of the 3.5 workbench. ______________________________________________________________________________ Theory of operation: PoolMem splits all available memory in two blocks: One block is used for the small memory snippets (always taken from there), the other block is used for huge allocations. This partition of the main memory is dynamic, i.e. each sub-pool can grow and shrink, depending on the memory requirements. The "public/ANY" memory gets a special treatment. PoolMem manages a "scratch" list for tiny memory blocks taken from there. Instead of taking these tiny memory blocks always from the main memory, they are taken from this (global, though) memory pool and put back into this pool if they get freed. A special garbage collection task cleans this "scratch list" from time to time, or if it overruns. The main profit is taken from the layers.library, which uses to allocate tons of tiny snippets and is therefore the main cause for memory fragmentation. The "chip" memory is treated a bit different. It's also split into two distinct memory pools (small and large), but memory from the large pool is allocated in reverse direction - unless you run PoolMem with the ChipFwd keyword, of course; this happens for two reasons: First, it helps to keep the memory defragmentated, so the big pool can't run that easely into the small pool. Second, it works around a hardware bug of my computer (the refresh of the high end chip memory in my computer seems to be a bit buggy - some bits tend to flip if they aren't frequently accessed, for example by the DMA processor as display memory.) For details about the PatchRAM program, check its readme file. As I said, if you run PoolMem, you're supposed to run PatchRAM as well. It helps PoolMem a lot in its job! Future versions of PoolMem will include this patch as well, but the current version is not yet supposed to be "final" in any way. _________________________________________________________________________________ Additional programs: The PoolMem program is still in a somewhat experimental stage, even though it's running stable for my system for more than two years now - I won't expect any serious bugs, though. However, if you like to see how PoolMem works and if it has any effects, I provided several extra programs: ShowMem: Shows the allocated/free memory in a graphical over- view. For details, check the ShowMem readme and its guide. (Available separately as well) PatchRAM: Modifies and improves the RAM disk. Shows the correct size of the RAM drive, improves the memory allocation sheme of it as well. Should be run together with PoolMem. For details, check the guide of PatchRAM. FragMeter: Calculates the fragmentation of your memory. The output is given separately for each memory type. A 100% fragmentation indicates that all the free memory is messed up in tiny blocks of eight bytes each (maximal fragmentation). A Shannon-type approach is used to measure the fragmentation (the algorithm calculates the Shannon entropy of the memory blocks with the formula sum += log(chunk->mc_Bytes/total) If you know a better approach for measuring the fragmentation, lemme now. This here seems at least reasonable for me as a theoretical physicst... ;-) This program can be used to test the efficency of PoolMem. My measurements indicate that the entropy is about halved. MemoryMess: A program that tries to fragmentate the main memory as bad as possible by allocating and freeing memory in random order. Can be used together with the FragMeter to measure the efficiency of PoolMem or with ShowMem to watch PoolMem at its job. Can be canceled safely with ^C (Control-C). Does nothing useful except that. PatchReplyMsg: Useful to find buggy programs, like XiPaint. PatchReplyMsg will create a guru 0x0100001f if a program attempts to reply an already de-allocated message. Don't install it permanently, but use it to find the reason for crashes. If you've ideas how to improve PoolMem, lemme know.... ______________________________________________________________________________ Guru meditations thrown by PoolMem: 0x01000013 Scratch entry illegal. Some program invalidated the internal memory scratch list of PoolMem - by overwriting memory that has been deallocated before. An (in)famous example is XiPaint. Run "PatchReplyMsg" and a debugger (e.g. COP) to check for details. 0x0100000f MemHeader not found. Some program attempted to free a non-existing block of memory. PoolMem (or exec) wasn't able to locate the MemHeader. 0x01000011 MemHeader insane. PoolMem found a MemHeader whose number of available bytes does not match the size of its pool. 0x01000012 Invalid DeleteHeader (internal). Someone tried to deallocate the "large" memory pool. This is an internal guru, shouldn't happen. 0x01000014 Memory Pools unordered. Someone changed the order of the memory blocks, the large memory block is no longer in front of the small memory pool. Additionally, PoolMem throws all standard exec guru's - the 1.3x routines replace the OS functions completely. As a side effect, PoolMem MUST BE RUN IN FRONT OF OTHER MEMORY TOOLS, especially RUN POOLMEM FIRST, MUNGWALL LATER. ______________________________________________________________________________ The THOR-Software Licence This License applies to the computer programs known as "PoolMem", "ShowMem", "FragMeter", "MemoryMess" and "ShowMem". The "Program", below, refers to such program. The programs and files in this distribution are freely distributable under the restrictions stated below, but are also Copyright (c) Thomas Richter. Distribution of the Program by a commercial organization without written permission from the author to any third party is prohibited if any payment is made in connection with such distribution, whether directly (as in payment for a copy of the Program) or indirectly (as in payment for some service related to the Program, or payment for some product or service that includes a copy of the Program "without charge"; these are only examples, and not an exhaustive enumeration of prohibited activities). However, the following methods of distribution involving payment shall not in and of themselves be a violation of this restriction: (i) Posting the Program on a public access information storage and retrieval service for which a fee is received for retrieving information (such as an on-line service), provided that the fee is not content-dependent (i.e., the fee would be the same for retrieving the same volume of information consisting of random data). (ii) Distributing the Program on a CD-ROM, provided that the files containing the Program are reproduced entirely and verbatim on such CD-ROM, and provided further that all information on such CD-ROM be redistributable for non-commercial purposes without charge. Everything in this distribution must be kept together, in original and unmodified form. Limitations. THE PROGRAM IS PROVIDED TO YOU "AS IS," WITHOUT WARRANTY. THERE IS NO WARRANTY FOR THE PROGRAM, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. IF YOU DO NOT ACCEPT THIS LICENCE, YOU MUST DELETE ALL FILES CONTAINED IN THIS ARCHIVE. ______________________________________________________________________________ Have fun, Thomas April 1998 ============================= Archive contents ============================= Original Packed Ratio Date Time Name -------- ------- ----- --------- -------- ------------- 21088 8777 58.3% 20-Apr-98 20:09:56 +Developer.readme 856 377 55.9% 11-Apr-98 15:42:36 +Developer.readme.info 5932 3947 33.4% 20-Apr-98 19:20:14 +FragMeter 1820 1290 29.1% 07-Mar-98 17:16:08 +MemoryMess 1140 800 29.8% 26-Apr-98 18:30:54 +PatchRAM 3547 1714 51.6% 26-Apr-98 18:30:50 +PatchRAM.readme 856 377 55.9% 11-Apr-98 15:42:36 +PatchRAM.readme.info 180 151 16.1% 25-Jan-98 13:35:00 +PatchReplyMsg 11416 5845 48.7% 26-Apr-98 18:52:14 +PoolMem 22144 8714 60.6% 26-Apr-98 18:46:14 +PoolMem.readme 856 379 55.7% 11-Apr-98 15:42:36 +PoolMem.readme.info 3672 2352 35.9% 19-Apr-98 17:54:50 +ShowMem 5009 2364 52.8% 19-Apr-98 17:54:44 +ShowMem.guide 537 284 47.1% 11-Apr-98 15:42:36 +ShowMem.guide.info 1110 379 65.8% 11-Apr-98 15:42:36 +ShowMem.info 906 453 50.0% 29-Mar-98 18:49:28 +ShowMem.readme -------- ------- ----- --------- -------- 81069 38203 52.8% 27-Apr-98 18:25:40 16 files